The present invention relates to a three-dimensional information processing apparatus and method for extracting three-dimensional information, that can be used in CG, CAD, and the like, from an object having a three-dimensional shape.
As a conventional technique for obtaining the three-dimensional shape of an object, for example, xe2x80x9cStereoscopic matching using a plurality of base line distancesxe2x80x9d (Journal of Papers of the Institute of Electronics, Information and Communication Engineers D-II, Vol. J75-D-II, No. 8, pp. 1317-1327, August 1992) is known. Generally, the conventional method of acquiring a three-dimensional shape can be roughly classified into passive and active methods.
One typical passive method is a stereoscopic image method, which utilizes trigonometric measurements using two cameras. In this method, the positions of images of an identical object are detected from right and left images taken by cameras, and the three-dimensional position of the object is measured based on the displacement amount between the detected positions.
As typical active methods, an optical radar type range finder which obtains distance by measuring the time until light projected toward and reflected by an object returns, a slit light projection method for projecting a slit-shaped light pattern onto an object, and measuring the three-dimensional shape on the basis of the displacement of the pattern shape formed on the object, and the like are known.
Note that the three-dimensional data of the object obtained by the above-mentioned methods can be reproduced and displayed on, e.g., a two-dimensional display.
However, the stereoscopic image method has as its major objective ;to calculate the distance information from a specific position where the cameras are set to the object, and does not measure the three-dimensional shape itself of a certain object. In the active methods, since a laser beam or the like must be irradiated onto the object, it is cumbersome to use such methods.
For this reason, such methods cannot flexibly cope with a dynamic image sensing environment, i.e., image sensing while moving around a certain object, and hence, none of the conventional methods can extract depth information in such dynamic image sensing environment.
Images normally used in an office are often finally output onto paper sheets, and images types to be used include both natural images and line images that express objects by edge lines alone. More specifically, in an office or the like, it is a common practice to process image information for various purposes.
In contrast to this, since the principal object of the above-mentioned prior art is to calculate the three-dimensional shape data of the object from certain specific setting positions of the cameras and to faithfully display the calculated data on a two-dimensional display, the above-mentioned methods cannot cope with various kinds of image processing required in, e.g., an office.
More specifically, the present invention is addressed to a three-dimensional information extraction apparatus which can be easily applied to a dynamic image sensing environment in which the image sensing position changes, and can process acquired three-dimensional information into various forms.
Some stereoscopic image processing apparatuses use three or more images in place of two images, and form three-dimensional shapes by unifying shape information obtained from such images.
Upon judging the reliability of the obtained three-dimensional shape, for example, the above-mentioned stereoscopic image method uses the comparison result or correlation of residuals obtained upon calculating the position displacement amount by corresponding point extraction of the luminance values in place of reliability judgment.
However, in the above-mentioned prior arts, in the case of, e.g., the stereoscopic image method, even when the residual is large or when the correlation function is small, if the angle the object makes with the image sensing plane is large or the distance from the apparatus to the object is large, calculation errors due to minimum errors of the corresponding extraction results are large, and the obtained three-dimensional shape has low reliability. On the other hand, the obtained three-dimensional shape is not displayed considering its low reliability.
That is, the present invention is also addressed to improvement of reliability in three-dimensional information processing.
On the other hand, the present invention is addressed to storage of image information in the dynamic image sensing environment. Problems associated with storage of image information in the dynamic image sensing environment will be discussed below.
In a certain prior art associated with the dynamic image sensing environment, a single image sensing unit placed on a rail is translated to sense a plurality of images, and shape analysis is made using the correlation among the sensed images.
In addition, Japanese Patent Publication No. 7-9673 is known as the technique of analyzing the shape of a stereoscopic object using the correlation among two pairs of parallax images sensed at the same time using a compound-eye image sensing device which is made up of a plurality of image sensing units. In this prior art, the image sensing device is fixed to a robot arm, and is moved as instructed to sense images.
A conventional image sensing apparatus which allows the photographer to freely carry the image sensing apparatus main body and can analyze the shape of an arbitrary object will be described below.
FIG. 1 is a block diagram showing the arrangement of a conventional portable automatic image sensing apparatus and the principle of its use state.
In FIG. 1, reference numeral 1101 denotes an object to be sensed (a cup in this embodiment), which is placed on a pad 1102, and a case will be explained below wherein this object 1101 is to be sensed. A plurality of bright point marks 1103a, 1103b, and 1103c are printed on the pad 1102, and their position relationship is known and is pre-stored in an image sensing apparatus 1900 (to be described below).
Reference numeral 1900 denotes a portable image sensing apparatus, which comprises photographing lenses 1110 and 1111, shutters 1112 and 1113 which also serve as iris diaphragms, image sensing elements 1114 and 1115 for performing photoelectric conversion, control circuits 1116 and 1117 for controlling the image sensing elements 1114 and 1115, image signal processing circuits 1118 and 1119 for processing signals obtained from the image sensing elements 1114 and 1115, image signal storage circuits 1120 and 1121 for storing image signals output from the image signal processing circuits 1118 and 1119, a corresponding point extraction circuit 1122, an image sensing parameter detection circuit 1123, a ROM (read-only memory) 1124 that stores the (known) position relationship among the bright points on the pad, a unifying circuit 1125 for unifying three-dimensional information, and buffer circuits 1126 and 1127 for temporarily storing the three-dimensional information unified by the three-dimensional information unifying circuit 1125.
This image sensing apparatus 1900 extracts corresponding points from the obtained two image signals by the corresponding point extraction circuit 1122 to obtain distance images at the individual timings, and at the same time, obtains image sensing parameters (the position relationship between the pad and the image sensing apparatus 1900 obtained based on the bright point coordinate positions, accurate focal length, and the like) using the image sensing parameter detection circuit 1123 and the ROM 1124. The three-dimensional information unifying circuit 1125 calculates three-dimensional shape data and texture image data of the object 1101 on the basis of these distance images, image sensing parameters, and change information that expresses their time-series changes, and stores them in the buffer circuits 1126 and 1127.
In FIG. 1, reference numeral 1140 denotes numerical value data of the three-dimensional shape of the object 1101 output from the image sensing apparatus 1900; and 1141, developed image data of the surface texture of the object 1101. These output data are transferred to a personal computer or the like, which performs texture mapping to display the input data as a stereoscopic CG (computer graphics) image. The display angle, size, and the like of the CG image can be instantaneously changed, and the image can also be deformed and processed. Two CG images which have slightly different view points are generated, and are output to a stereoscopic display, thus allowing the user to observe a stereoscopic image. In this case, since the stereoscopic image can be freely rotated and deformed, the user can experience higher reality.
In the image sensing apparatus 1900, the corresponding point extraction circuit 1122 and the three-dimensional information unifying circuit 1125 require the most complicated, time-consuming processing and, hence, require a very large circuit scale and consumption power. The image sensing apparatus 1900 has a sequential processing mode in which such complicated processing is sequentially executed while sensing images, and a simultaneous processing mode in which the required sensed images are stored in the image signal storage circuits 1120 and 1121, and thereafter, the processing is executed simultaneously. On the other hand, the image sensing apparatus 1900 allows the photographer to freely carry the image sensing apparatus 1900 without requiring any large-scale positioning device unlike in the above-mentioned prior art, and can easily analyze the shape of the object 1101 without requiring any special preparation processes.
However, the prior art shown in FIG. 1 suffers the following problems.
More specifically, in general, accurate positioning cannot be attained at a constant speed even by the operation of the photographer unlike in the above-mentioned conventional positioning device. For example, when images are stored in the image storage circuit at given time intervals and are subjected to image processing, redundant information increases in a portion sensed by moving the apparatus at an excessively low speed, and a very large image memory capacity is required, resulting in a long shape analysis time. Furthermore, the analyzed three-dimensional shape data becomes excessively fine, and the subsequent CG generation requires an extra processing time and storage capacity. Conversely, when the photographer moves the image sensing apparatus at high speed, information required for analyzing the three-dimensional shape becomes short, and the analysis precision is impaired. In the worst case, if an image of a specific side surface of the object cannot be acquired, the shape information of that portion is lost.
The present invention has been made in consideration of the above situation, and has as its object to provide a three-dimensional information processing apparatus and method, which can flexibly cope with dynamic image sensing, and can process the obtained three-dimensional information into various forms.
In order to achieve the above object, according to the present invention, there is provided a three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:
image sensing means for sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems;
information extraction means for extracting a plurality of depth information from image sensing related information sensed by the image sensing means at the plurality of coordinate positions; and
conversion/unification means for converting and unifying the plurality of depth information extracted by the depth information extraction means into depth information expressed by a unified coordinate system.
Also, in order to achieve the above object, according to the present invention, there is provided a three-dimensional information processing method for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:
the first step of sensing images of the object from a plurality of coordinate positions using an image sensing system having one or a plurality of optical systems;
the second step of extracting a plurality of depth information from image sensing related information sensed at the plurality of coordinate positions in the first step; and
the third step of converting and unifying the plurality of depth information extracted by the depth information extraction means into depth information expressed by a unified coordinate system.
According to the apparatus and method with the above arrangement, upon unifying depth information, since a plurality of depth information are converted into depth information expressed by a unified coordinate system on the basis of, e.g., the luminance information of the object and displacement information of distance information, the present invention can flexibly cope with dynamic image sensing in which image sensing is done while moving the apparatus around a certain object, and can easily process the obtained information into various image forms.
According to one preferred aspect of the present invention, a displacement between coordinate systems of the plurality of depth information is detected on the basis of the image information of the object.
According to one preferred aspect of the present invention, the unified coordinate system has five different projection planes.
According to one preferred aspect of the present invention, the image information includes luminance information of the object, and the displacement between the coordinate systems is detected on the basis of the luminance information.
In order to achieve the above object, according to the present invention, there is provided a three-dimensional information processing apparatus for obtaining three-dimensional information from an object having a three-dimensional shape, and performing predetermined information processing, comprising:
image sensing means for sensing images of the object using an image sensing system having one or a plurality of optical systems;
three-dimensional shape extraction means for extracting three-dimensional shape information of the object from image sensing related information sensed by the image sensing means; and
reliability determination means for determining reliability of the three-dimensional shape information extracted by the three-dimensional shape extraction means.
It is another object of the present invention to provide a three-dimensional information processing apparatus and method, which can notify the discrimination result of reliability.
It is still another object of the present invention to provide a three-dimensional information processing apparatus and method, which can process three-dimensional shape information in accordance with the discrimination result of reliability, and can display the processed three-dimensional shape information.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of an angle of the object with respect to an image sensing plane.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of a distance between the image sensing means and the object.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of an angle a pad that places the object thereon makes with an image sensing plane of the image sensing means.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of an area ratio of a pad that places the object thereon to an image sensing region.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of a position of a pad that places the object thereon.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of reflected light information reflected by the object.
According to one preferred aspect of the present invention, the reliability of the three-dimensional shape information is determined on the basis of a degree of correspondence of pixels between a plurality of image sensing related data sensed by the image sensing means.
It is still another object of the,present invention to provide an image sensing method and apparatus, which can minimize the storage capacity of storage means that stores images, can shorten the time required for processing images, and can avoid any errors upon executing processing or display after image sensing.
In order to achieve the above object, according to the present invention, there is provided an image sensing method comprising:
the image sensing step of sensing images of an object;
the storage step of storing image information of the object;
the image sensing condition detection step of detecting a relative relationship between the object and an image sensing apparatus main body; and
the control step of controlling a storage operation of the image information,
wherein the control step includes the step of controlling the storage operation in the storage step in accordance with a detection result of the image sensing condition detection step.
Also, in order to achieve the above object, according to the present invention, there is provided an image sensing apparatus comprising:
image sensing means for sensing images of an object;
storage means for storing image information of the object;
image sensing condition detection means for detecting a relative relationship between the object and an image sensing apparatus main body; and
control means for controlling the storage means,
wherein the control means controls the storage means in accordance with an output from the image sensing condition detection means.
According to the method or apparatus with the above arrangement, since the required minimum capacity of images used in image display and three-dimensional shape analysis is always stored, the storage capacity of the storage means can be reduced, and the operation time of the three-dimensional shape analysis processing means can be shortened, thereby realizing a size reduction and a cost reduction of the overall apparatus.
In order to achieve the above object, according to the present invention, there is provided an image sensing method comprising:
the image sensing step of sensing images of an object;
the analysis step of analyzing image information obtained in the image sensing step;
the image sensing condition detection step of detecting a relative relationship between the object and an image sensing apparatus main body; and
the control step of controlling an image analysis operation in the analysis step,
wherein the control step includes the step of controlling the image analysis operation in accordance with a detection result of the image sensing condition detection step.
Also, in order to achieve the above object, according to the present invention, there is provided an image sensing apparatus comprising:
image sensing means for sensing images of an object;
image analysis means for analyzing image information sensed by the image sensing means;
image sensing condition detection means for detecting a relative relationship between the object and an image sensing apparatus main body; and
control means for controlling the image analysis means,
wherein the control means controls the image analysis means in accordance with an output from the image sensing condition detection means.
According to the image sensing method and apparatus with the above arrangement, since required minimum images alone are subjected to three-dimensional shape analysis processing, the operation time of the three-dimensional analysis can be shortened, and loss of required images can be avoided, thus realizing a size reduction and a cost reduction of the overall apparatus.
According to one preferred aspect of the present invention, control is made to store information associated with the relative relationship between the object and the image sensing apparatus main body together with sensed images sensed in the image sensing step in the storage step. The stored information can be easily compared with desired observation direction information input by the observer upon reproduction of an image, and an appropriate image can be instantaneously displayed.
According to one preferred aspect of the present invention, the image sensing condition is detected using a sensor for detecting an angle and translation movement of the image sensing apparatus main body. Sampling positions can be assigned on the space at nearly equal intervals by a simple apparatus arrangement.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting an angle and translation movement of the image sensing apparatus main body on the basis of changes in state of sensed images sensed in the image sensing step. The sampling interval of images can be appropriately changed in correspondence with the complexity of the object structure.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image and images around the object sensed by the image sensing apparatus main body, and detecting changes in relative position relationship between the object and the image sensing apparatus main body on the basis of an error signal generated upon analyzing the images. Since the shape information of the object region that could not be analyzed at a certain time can be compensated for using information obtained by analyzing an image at a different time, accurate three-dimensional shape data can always be output.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in occlusion state of the object. Even for an object with a complicated shape, regions that cannot be analyzed are few, and accurate information can be output as a whole.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting an overlapping region area between time-serial object images. In particular, when high-magnification image sensing is done, joint analysis between images can be performed from images with predetermined precision, and loss of required images can be avoided.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in distance image of the object. In the object region corresponding to a complicated three-dimensional shape, the number of times of sampling can be increased, and high-precision three-dimensional shape data can be output.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of analyzing an object image sensed by the image sensing apparatus main body, and detecting changes in distance image of the object. In the object region corresponding to a complicated three-dimensional shape, the number of times of sampling can be increased, and high-precision three-dimensional shape data can be output.
According to one preferred aspect of the present invention, the image sensing condition detection includes the step of stopping the image sensing step and the analysis step during a period in which neither storage processing nor analysis processing are performed. Since the image sensing means and the image analysis means that consume relatively large power cease to operate during the period that requires neither image storage nor processing, consumption power can be greatly reduced.
According to one preferred aspect of the present invention, the image analysis step includes the step of performing an analysis calculation for acquiring a three-dimensional shape and a surface image of the object using a plurality of images. Accordingly, since an object image is generated using texture mapping or the like in computer graphics, the observer can freely select the observation direction and distance, and the three-dimensional shape and surface state.